Cryptography based on neural networks—analytical results

The mutual learning process between two parity feed-forward networks with discrete and continuous weights is studied analytically, and we find that the number of steps required to achieve full synchronization between the two networks in the case of discrete weights is finite. The synchronization process is shown to be non-self-averaging and the analytical solution is based on random auxiliary variables. The learning time of an attacker that is trying to imitate one of the networks is examined analytically and is found to be much longer than the synchronization time. Analytical results are found to be in agreement with simulations. PACS numbers: 87.18.Sn, 89.70.+c The study of neural networks was originally driven by its potential as a powerful learning and memory machine. Statistical mechanical methods have been used to analyse the network’s ability and explore its limitations [1, 2]. In a recent paper [3], a bridge between the theory of neural networks and cryptography was established. It was shown numerically that two randomly initialized neural networks with one layer of hidden units (so-called parity machines (PMs)[4]) learning from each other, are able to synchronize. The two parties have common inputs and they exchange information about their output. In the case of disagreement, the two PMs are trained by the Hebbian learning rule on their mutual outputs and they develop a full synchronized state of their synaptic weights. This synchronization procedure can be used to construct an ephemeral key exchange protocol for the secure transmission of secret data. An attacker who knows the architecture of the two parties, the common inputs, and observes the mutual exchange of information, finds it difficult to imitate the moves of the parties and to reveal the common parameters after synchronization. All parties have secret information which is not known either to other members or to possible attackers: their initial weights and the current state of their hidden units are noted as internal representations (IRs). In most applications a public-key system is used which is based on number theory where the keys are long integers, and the complexity of the encryption/decryption processes scales polynomially with the size of the key [5]. In this letter we present a cryptosystem which is 0305-4470/02/470707+07$30.00 © 2002 IOP Publishing Ltd Printed in the UK L707 L708 Letter to the Editor based on biological ideas including the network architecture, biological operations and the learning process, and the complexity of the generation of the secure channel is linear with the size of the network. This biological mechanism, which is shown analytically to be robust against a possible attack, may be used to construct an efficient encryption system using keys which change permanently. During the last decade, the analysis of learning from examples performed by feed-forward multi-layered networks was exhaustively examined using statistical mechanical methods [1, 2]. The study of the generalization ability of such networks was based on a set of training examples generated by a static teacher network. Here we discuss a case where two or several multilayer networks are trained by their mutual outputs. This scenario has been solved only for perceptrons with continuous weights [6]. Here we present an analytic solution for PMs with continuous as well as with discrete weights. In our cryptosystem, each party in the secure channel is represented by a feed-forward network consisting of KN random input elements xji = ±1, j = 1, . . . , N,K binary hidden units τi = ±1, i = 1, . . . ,K and one binary output unit σ = iτi . For simplicity of calculations presented below we concentrate only on the case of a tree PM with three binary hidden units feeding a binary output σ = τ1τ2τ3. The hidden units are determined via Boolean functions τi = sgn (∑ j Wjixji ) through three disjointed sets of inputs Xi = x1i , . . . , xNi. The weights are either discrete or continuous, and the analytical results are derived for N 1. In this letter we present: (a) an analytical solution of the mutual learning of two PMs whose weight-vectors are updated according to the mismatch between their mutual information—their outputs. Synchronization is achieved in the case of discrete weights, Wji = 0,±1, . . . ,±L, as well as for continuous weights confined to a sphere, ∑N j=1 W 2 ji = N . (b) Analysis of online adaptation of discrete weights, in which each change of a component is not infinitesimally small, demands different methods than the standard ones [7], and this is at the centre of the discussion below. Surprisingly, synchronization is achieved for the discrete weights at a finite number of steps. (c) Dynamical evolution of the discrete networks cannot be characterized by the time evolution of the standard order parameters, since the overlaps between the weightvectors are not self-averaging [8] even for large networks. The analytical solution is based on calculation of the evolution of the distribution of the order parameters as a function of the initial set of the weights. (d) The analysis is extended to include a possible attacker. For simplicity of presentation, we first describe the analytical methods developed for the discrete case where detailed results are presented for particular examined cases. At the end of this letter results for the continuous case are also briefly summarized. The definition of the updating procedure between the two parties, A and B, that are trying to synchronize their weights, is as follows. In each time step, the output of the two parties is calculated for a common random input. Only weights belonging to the one (or three) hidden units which are equal to their output unit are updated, in each of the two parties. The updating is done according to the following Hebbian learning rule, W ji = W ji + K ( W jixjiσ B ) xjiσ θ ( στ i ) θ(−σAσB) W ji = W ji + K ( W jixjiσ A ) xjiσ θ ( στ i ) θ(−σAσB) (1) where K(y) = 1− δL,y and δ represents the Kronecker function. The purpose of the operator K(y) is to prevent the increment (decrement) of the strength of the weights on the boundary value L(−L). Two important simulation results are crucial for the analytical description of the mutual dynamics. The first observation is that the synchronization time is finite [3]. The second is that different runs (set of random inputs) of the above dynamics, but with fixed initial conditions for the two parties, result in different sets of IRs. As a result of these two observations, we Letter to the Editor L709 realized that the variance of the overlaps between the two parties is finite and does not shrink to zero even in the thermodynamic limit. This unusual scenario of online mutual learning is taken into consideration in the analytical equations, by the selection of random IRs following the freedom given by the current analytical overlaps. We find an iterative discrete set of equations for the mutual overlaps between the parties, whose evolution depends on some random but correlated ingredients—the current IRs, { τ i } , { τ i } (see equation (1)). In each time step, μ, the mutual state of the two parties is defined by a (2L+ 1)× (2L+ 1) matrix, F i(μ), where i represents the hidden unit. The element f i qr of the matrix stands for the fraction of components in the ith weight-vector which is equal to q(r) in the first(second) party, where q, r = 0,±1, . . . ,±L. The overlap of the weights belonging to the ith hidden unit in the two parties, R i = WAi · WBi / N , as well as their norms, Qi = Wi · Wi/N, are given by the matrix elements